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AfiSTRACI 

As part of a project designed to identify appropriate 
and feasible sethods of evaluating vocational education, this essay 
provides a perspective for evaluating vocational education in terms 
of outcoaes. The first of four sections provides a rationale for 
studying educational outcones. Section 2 focuses o^ the nature and 
meaning of ^outcomes and the range and diversity of vocational 
education cutcomes, and outlines an approach to outcomes evaluation 
within the general framevcrh of vocational education evaluation. In 
addition, this section provides a list of thirty guestions concerning 
possible outcomes of vocational education which illustrates the broad 
and diverse range of possible outcomes. Among the key Issues 
'identii:ied in section 3 in the area of outcomes evaluation were (1) 
the definition of vocational education; (2) the specification of 
vocational programs as educational treatments; and (3) sources and 
uses of evaluative criteria a fid experimental controls, and the use of 
comparison groups in evaluating vocational educatict programs with 
respect to specified outcomes. The final section suggests an agenda 
for improving outcome evaluation. (LRA) 
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FOREWORD 



Educational outcomes have commanded increased attention 
in recent years as the focus of evaluation has shifted from 
processes to products and impact. Are vocational education 
programs achieving their goals? What are the side effects of 
vocational education? What are the tangible payoffs to indivi- 
duals and society from resources invested in vocational educa- 
tion? 

This essay provides perspective for evaluating vocational 
education in terms of outcomes. The author clarifies the 
nature and meaning of outcomes, expands our awareness of the 
range and diversity of vocational education outcomes, outlines 
an approach to outcomes evaluation within the general frame- 
work of vocational education evaluation, identifies some issues 
and problems relative to Mtcomes evaluation, and suggests an 
agenda for strengthening research and evaluation in the field 
of vocational education outcomes. 

Audiences that may find this publication especially use- 
ful include the research, development, and evaluation (RD&E) 
community in vocational education and training; policy analysts 
and educational leaders; and some pract ' tioners in state and 
local education agencies. 



Vocational Education Outcomes; Perspective for Evaluation 
originally appeared under the tiv.le "Examining Vocational Edu- 
cation Outcomes: Perspective on che State of the Art," Attach- 
ment "C" in Vocational Education Outcomes , Final Report on Year 
One of the R&D Project "Examining Vocational Educational Outcomes 
and Their Correlates." The report was prepared at the National 
.Center by Robert L. Darcy, Kathleen A. Bo Hand, and Joanne Farley 
with the assistance of Carolyn M. Taylor, and submitted to the 
U.S. Office of Education in January 1979. Other coirponents of 
that report will also be published as separate documents with 
the titles Vocational Education Outcomes: A Thesaurus of Outcome 
Questions ; Vocational Education Outcomes: An Evaluative Bibll^ 
ography of Empirical Studies ; and Vocational Education Outcomes: ; 
Annotated Bibliography of Relatea Literature . ^ 

In the development of the general report and i^ various 
components, a number of people made valuable contributions. 
Distinquished representatives from business, labor, and educa- 
tion who served on the National Workshop Panel on Vocational 
Education Outcomes were: Walter G. uavis, American Federation 
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of Labor and Congress of Industrial Organizations? Mary Ellen 
Hillaire, Evergreen State College (Washington); Addison S. 
aobbs, Michigan Department of Education (now with the Maryland 
Department of Education) ; 0. Louise Lothspeich, Oregon State 
Advisory Council on Vocational Education; Fred D. Mack, Central 
State University (Oklahoma); Gerald Q. Miller, Sharon Steel 
Corporation (Pennsylvania) and Employers National Job Service 
Improvement Committee; Philip L. Smith, The Ohio State Univer- 
sity; Jerry C. Olson, Pittsburgh Public Schools (Pennsylvania) ; 
Wallis E. Pereira, Industry Education Council of California; 
Clio S. Reinwald, Arizona Department of Educ .tion; Annell L. 
Simcoe, Rutgers University; and J. Robert Warmbrod, The Ohio 
State University. 

We are also' grateful to Michael D. Hock of Worthington, 
Ohio, Douglas Sjogren of Colorado State University, and Michael 
Scriven of the University of San Francisco, who served as con- 
sultants for the study; to the five members of the National 
Cf»nfpr*s Evaluation Technical Advisory Panel — Carol B. Aslanian, 
College Entrance Examination Board; George C. Copa, University 
of Minnesota; Donald W. Drewes, Conserva Inc.; Ruth P. Hughes, 
Iowa Stats University; and Daniel L. Stufflebeam, Western Michi- 
gan University — and to Jo Ann M. Steiger of Steiger, Fink & 
Kosecoff, and David J. Pucel of the University of Minnesota, as 
well as T. Lynne Peterson and William L. Hull of the National 
Center staff, for reviewing drafts of the full report. 

On behalf of the National Center, I want to express appre- 
ciation to the Bureau of Occupational and Adult Education, ' .S. 
Office of Education, for sponsoring this study and to Robert L. 
Darcy, Senior Research Specialist at the National Center both 
for directing the overall project and preparing the present 
esiay . 

Robert E. Taylor 
Executive Director 
The National Center for Re- 
search in Vocational 
' Education 
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VOCATIONAL EDUCATION OUTCOMES: 
PERSPECTIVE FOR EVALUATION^ 



!• tNTRODUCTIOM 

This paper deals with the theory and practice of voca- 
tional education outcomes evaluation, which is part of the 
broader field of educational evaluation. It is not intended 
as a review and synthesis paper or comprehensive state-of- 
the-art essay on vocational education evaluation but mainly 
as a contribution to the conceptual basis of outcomes analysis. 



A. Purpose of the Essay 

The overall purpose of the essay and the study to which 
it relates is to help answer two research questions: (1) What 
particular outcomes are appropriate for use as criteria to 
evaluate vocational education programs? and (2) What procedures 
can be used for evaluating vocational programs in terms of 
these outcomes? 

Special attention in this paper is focused on: 

— clarifying the nature and meaning of vocational educa- 
tion outcomes; 

— expanding awareness of the range euid diversity of out- 
comes ; 



This essay was prepared as part of The National Center's 
applied research and development project, "Examining Voca- 
t'^nal Education Outcomes and Their Correlates," under a con- 
tract with the Bureau of Occupational and Adult Education, 
Office of Education, U.S. Department of Health, Education, and 
Welfare. Other products of this study include an evaluative 
bibliography of vocational education outcome studies, a the- 
saurus of vocational education outcome questions, and an an- 
notated bibliography of literature related to vocational 
education outcomes. The author wishes to express appreciation 
to John T. Grasso, Michael Scriven, Douglas Sjogren, and 
Daniel L. Stuff lebeam for their review of an earlier draft, 
and his special thanks to Kathleen A. Bol land, Joanne Farley, 
Carolyn M. Taylor, and other colleagues at the. National Center 
for their helpful comments and suggestions concerning the paper 
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—developing an approach to outcomes evaluation within the 
general framework of vocational education evaluation; 

—identifying some issues, problems, and practices in the 
evaluation of vocational programs with respect to outcomes; 
and 

—suggesting an agenda for strengthening research and eval- 
uation in the field of vocational education outcomes, 
which in turn can contribute to improving the effective- 
ness and efficiency of educational programs. 

The present paper does not identify a blue-ribbon list of 
valid, operational outcomes nor does it lay out a procedure for 
doing so. Those tasks remain to be accomplished in the future. 

The audience addressed by this paper includes educational 
researchers and evaluators, vocational education policy makers, 
academics, and the general public. While some of the ideas 
and information contained in this paper may be directly useful 
to practitioners, it is not intended as a how-to-do-it hand- j 
book for program managers, evaluators, or classroom teachers. 

B. Rationale for Studying Educational Outcomes 

Outcomes analysis is important to observers of vocational 
education for at least three very practical reasons • First, 



To help meet the needs of practitioners, the National 
Center has developed Evaluation Handbook: Guidelines and 
1>ractices for Fol low-Up Studies of Former Vocational Education 
Students (Stephen J. Franchak and Janet Weiskott, 1978). The 
bibliography alluded to in footnote 1 reports on design and 
procedures that have been used in some empirical outcome 
studies conducted during the past decade (see Kathleen A. Hol- 
land, Vocational Education Outcomes: An Evaluative Bibliography 
of Empirical Studies ; Columbus: National Center for Research 
in Vocational Education, The Ohio State University, 1979). Another 
Center project completed in 1978 is Interpreting Outcome Mea- 
sures in Vocational Education: A Final Report (Floyd L. Mc- 
Kinney, Kenney E. Gray, and Marie Abram) . Scheduled for pub- 
lication by the National Center in 19 79 are five state-of-the- 
art papers dealing with various aspects of evaluation: Needs 
Assessment for Program Planning in Vocational Education (Infor- 
mation Series No. 154); Longitudinal Methods as Tools for Eval- 
uating Vocational Education (IS No. 155); Use of Evaluative 
Data by Vocational Education (IS No. 156); Impact Evaluation in 
Vocational Education: The State of the Art (IS No. 157); and 
Data Sources for Vocational Education Evaluation (IS No. 158) . 
In addition, the National Institute of Education is producing 
papers and reports as part of its study and evaluation of voca- 
tional education mandated by the Education Amendments of 1976. 
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the states are mandated by Congress to evaluate program ef- 
fectiveness on the basis of tv^ specific outcome criteria: 
(a) employment in training-related occupations, and (b) employ- 
er assessment of the training and preparation for employment 
received by students. Man^more outcomes, such as educational 
achievement and human development, are perceived by educators 
and others as being of imme^nse importance both to individual 
students and society. Se^nd, public concern over tax bur- 
dens, rising costs of goyernment, and accountability regaftrd- 
ing returns from educatipnal spending have led to a search for 
tangible indicators of /program impact. Third, as educational 
evaluation has developed and matured, study of the broad area 
of outcomes and impact has assumed increased importance in a 
comprehensive program of evaluative research alongside such 
non-outcome variables as institutional context, student char- 
acteristics, resources,! program goals, and educational pro- 
cesses. I 

Moreover, these practical considerations are in keeping 
with the fundamental purposes of scientific inquiry: to dis- 
cover (1) the consequences of given actions (including pro- 
grams, policies, and institutional arrangements) and (2) the 
specific actions that will produce given consequences. In 
the policy sciences, there is a third function of research, 
namely to generate information which can help decision makers 
select the goals that will be treated as desirable "given" 
consequences and provide evaluators with standards to be used 
in evaluating observed consequences. In the present context, 
we are addressing conceptually, a question of "positive anal- 
ysis" — What are the various possible consequences of vocational 
education programs? and a question of "normative analysis" — 
Which of these consequences are appropriate and feasible to 
be used as criteria for evaluating vocational programs?^ 



•^U.S. Public Law 94-482, Education Amendments of 1976, 

Sec. 112(b). 

4 

Nobel prize winner Milton Friedman quotes John Neville 
Keynes in distinguishing among "positive science," a system- 
ized body of knowledge concerning what is; "normative science," 
knowledge addressing criteria of what ought to be; and "art," 
a system of rules for the attainment of a given end. Essays 
in Positive Economics ^Chicago: University of Chicago Press, 
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II. EDUCATIONAL EVALUATION AND OUTCOMES ANALYSIS 



A review of literature in the broad area of evaluation, 
the narrower field of educational evaluation, and the more 
specialized fieljd of vocational 'education evaluation discloses 
considerable variation in how people view the nature, pur- 
pose, and preferred methods of assessment and evaluation.. 



A. Assessment, Evaluation and Research 

In the Encyclopedia of Educational Evaluation , assess- 
ment is defined as ^' the process of gathering data and fashion- 
ing them into an interpretable form."^ Worthen and Sanders 
in Educational Evaluation: Theory and Practice identify five 
"schools of thought about how evaluation ought to be defined." 
As practiced by their respective adherents, these approaches 
are characterized as: (1) "educational measurement," (2) 
"professional judgment," (3) "comparing performance data with 
clearly specified objectives,," (4) "collecting information to 
assist decision-makers in choosin-' among available decision 
alternatives," and (5) "systematic collection and analysis 
of information to determine the worth of a thing." Irrespec- 
tive of approach, description is acknowledged to be a basic 
component of the evaluation process.^ Evaluation is sometimes 



^Sometimes the term "assessment" is used interchangeably 
with "measurement" and with "evaluation." See Scarvia B. Ander 
son and others. Encyclopedi a of Educational Evaluation (San 
Francisco: Jossey-Bass, I9T5) p. 27. 

^Blaine R. Worthen and James R. Sanders, Educational 
Evaluation: Theory and Practice (Worthington, Ohio: Charles 
Jones, 1973), pp. 20t, ioyt, 125. In their treatment of de- 
scription as a component of formal evaluation, Worthen and 
Sanders focus on the ideas of Robert E. Stake. More recently, 
writing in the newsletter of The Evaluation Network, Joe B. 
Hansen emphasizes the importance of description in program 
evaluation and laments the bias which evaluators display for 
experimental studies to the neglect of full and accurate de- 
scription. "With the advent of modern statistical techniques 
and the wide availability of computers to perform statistical 
analysis," he writes, "educational research and evaluation 
have become associated with statistical methodology to the 
point of eclipsing more fundamental considerations. It could 
be interpreted by practitioners of this emerging art-science 
as a sign of insecurity — (footnote continued on next page) 



referred to as evaluative (or evaluation) research . Scriven 
suggests that "the basic distinction [between * straight* re- 
search and evaluation reiiearch] seems to be that evaluation 
research must produce as a conclusion exactly the kind of 
statement that social scientists have for years been taught 
is illegitimate: a judgment of value, worth, or merit."* In 
the present paper we are adopting the Worthen and Sanders/ 
Scriven definition of evaluation as the determination of the 
worth of a, thing. In this view, evaluation involves (a) ob- 
servation and description of what is, and (b) a value judg- 
ment concerning what should be , i.e., appraisal or judgment 
of worth. — ^— ^ 

However defined, evaluation has numerous purposes and 
methods. The former include accountability, 'Compliance, pro- 
ject monitoring, program improvement, and providing useful 
information for judging decision alternatives such as whether 
to continue, expand, modify, or terminate programs.® Needs 
assessments are conducted, and both formative and summative 
evaluations are carried out. The diversity of evaluation 
methods is suggested by some of the terms one encounters in 
the literature: goal-based evaluation, goal-free evaluation, 
formal, informal, objective, subjective, process, on-site, 
product, follow-up, impact, payoff evaluation, benefit-cost 
studies, and cost-effectiveness analyses,., among others. 

This kaleidoscope of concepts, purposes, perceptions, and 
procedures adds a richness to evaluation that may, however, 
exact a toll. The other side of the diversity coin in educa- 
tional, evaluation is a lack oS consistency, clarity, and 



(continued) — that evaluators are often more concerned 
with the statistical significance of their results than with 
the importance of the questions to which the analyses are 
addressed." See "The Formative-Summative Dichotomy and the 
Role of Description in Program Evaluation," Evaluation News , 
No. 6, August 1978, pp. 14-16. 

7 

Michael Scriven, "Evaluation Perspectives and Proced- 
ures," pp. 3-93 in W. James Popham (ed.) Evaluation in Educa - 
tion (Berkeley: McCutchan, 1974), p. 4. 

q 

In Educational Evaluation and Decision Making (Itasca, 
111.: Peacock, 1971), prepared for the education society 
Phi Delta Kappa by Daniel L. Stuff lebeam and others, evalua- 
tion is defined as "the process of delineating, obtaining, 
and providing useful information for judging decision alter- 
natives." 
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credibility with respect to the knowledge base we have accum- 
ulated in the field of educational outcomes. ^ 
B, The Education Enterprise and Educati onal Evaluation 

It will be helpful to view educational outcomes and out- 
comes evaluation In the broader framework of the overall ed- 
ucation enterprise. 



5 A 1976 report by the Committee on Vocational Education 
Research and Development (COVERO) noted that "$250 million 
spent by the U. S. Office of Education In vocational education 
research and development during the last 10 years has not had 
documented, widespread Impact" ( Assesslncr Vocational Education 
Research and Development , National Academy of Sciences, p. 1) . 
Commenting on COVERD's findings, the president of the Ameri- 
can Vocational Education Research Association observed that 
"the substance and process of vocational education research 
leaves something to be desired" (J. Robert Warmbrod, "Evalua- 
tion Research In Vocational Education," paper presented at 
the AVERA annual meeting In Houston, Texas, December 6, 1976, 
and reprinted In Beacon , Vol. 6, No. 1 (January 1977) pp. 1» 
6-11.) Other evaluators have noted that "most of . . . [the 
evaluation of investments In the human agent, such as voca- 
tional and technical education) Is of little or no use for 
purposes of making reasoned choices among competing social 
programs . . due to the methodological errors which mar 
many studies as well as the lack of effective techniques to 
handle many of the more difficult Issues which arise as a part 
of the evaluative process (Jerome Moss, Jr., and Ernst W. 
Stromsdorfer, "Evaluation Vocational and Technical Education 
Programs" In Gerald G. Somers and J. Kenneth Little, Vocational 
Education: Today and Tomorrow , Center for Studies in Voca- 
tional and Technical Education, University of Wisconsin, 1971, 
p. 221) . Floyd L. McKlnney found "a lack of consistency In 
the use of acceptable procedures In conducting follow-up studies" 
( Program Evaluation In Vocational Education: A Review , ERIC 
Clearinghouse on Career Education, The Center for Vocational 
Education, Ohio Statr University, 1977, p. 24). Writing In 
an official U. 3. Department of Labor magazine In 1974, a 
Columbia University researcher observed that "the large body 
of studies (analyzing] the outcomes of high school vocational 
education [are] beset with conceptual problems, methodological 
pitfalls, and statistical limitations .... Proceeding 
from different values and assumptions, the analysts have not 
even agreed on the objectives or outcomes to be tested" (Beatrice 
G. Ruebens, "Vocational Education: Performance and Potential," 
Manpowe u ly 1974, pp. 23-29). For additional information 
on the state of vocational education evaluation see Darcy, 
Robert L. ; Holland, Kathleen A.; and Farley, Joanne. 
Vocational Education Outcomes (Columbus: National Center for 
Research in Vocational Education, The Ohio State University, 
1979) pp. viii and passim ^ well as Holland, Evaluative Biblio- 
graphy , passim ) . 



o 

ERIC 



Figure 1 lists six components of the educational system, 
each of which may be seen as an appreciate object for de- 
scription and evaluation. THese interdependent sets of edu- • 
cation variables are: 

(1) Context — the milieu of circumstahces and needs, 
organizational arrangements, societal values, insti<- 
tutional settings, and the physical environment with- 
in which education takes place; 

(2) Students — characteristics of the potential learners, 
who are recipients of investment activities and whose 
human capital Will be enhanced when the investment 
activities are successfully carried out; 

(3) Resources — in money terms, the funds allocated to 
pay for real resources employed in educational pro- 
duction; in real terms, the quality and quantity of 
human and physical factors of production so employed 
(e.g., teachers, physical plant, instructional sup- 
plies and equipment) ; 

(4) Goals — the stated or unstated aims of educational 
programs (i.e., intended educational . outcomes) ; and 

(5) Processes — instructional methods, progreun proced- 
ures^ and educa^^tional technologies (broadly defined); 

(6) Outcomes — short-term consequences and longer-term 
impact resulting from vocational programs (i.e., 
dependent variables associated with the interaction 
of the five preceding sets of independent variables) . 

This six-factor analytical framework is similar to the CIPP 
model (Context, Input, Process, Product) ' described in evalu- 
ation literature? although the terms are not defined in 



Acquired capabilities that can be used in one's capac- 
ity as a factor of production, i.e., as a human resource. 
See Robert L. Darcy, "The Nature of Economic Enterprise," in 
Edwin L. Herr (ed.) Vocational Guidance and Human Development 
(Boston: Houghton Mifflin 1974) , p. 121. Note. that human 
capital does not denote people, but capabilities embodied in 
people. For a simplified explanation of human capital invest- 
ment, see William E. Becker, Jr., "Investment in Human Capi- 
tal," pp. 59-76 in Warren G. Meyer (ed.) Vocational Education 
and the Nation's Economy (Washington, D. C: American Voca- 
tional Association, 1977) . 
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precisely the same way. It also resembles a conventional 
education production function, which relates input factors 
to educational products or outputs. ■^^ The five "input" fac- 
tors listed i i Figure 1, which interact to constitute educar 
tional programs or "treatments," are recast in Figure 2 (note 
input factors ni . . . n^) in the format of a production func- 
tion. This analogy suggests that over a period of time, and 
subject to the influence of intervening forces and error fac- 
tors, it is theoretically possible to observe certain outcomes 
as having been generated by (i.e., significantly associated 
with) given, educational treatments. 

As suggested above, each component of the. educational 
enterprise represents a potential obn^ect of systematic inquiry 
and evaluation. For purposes of description and appraisal, one 
may single out student characteristics (e..g., age, sex, men- 
tal ability, socioeconomic background of vocational students) ; 
or the quality of resources employed (e.g. , training and ex- 
perience of teacher^); or educational outcomes (e.g., skill 
proficiency, employment status, hourly earnings) for special 
scrutiny rather than addressing, under the rubric of "program 
evaluation," the totality of factors relevant to educational 
performance. 'Indeed, this selective, "partial equilibrium" 
methodology, based on the principles of specialization 'and 
division of labor, might produce the same results for vocational 
education evaluation that are claimed in economics : sharply 
increased productivity. We need to know far more than we pre- 
sently know about student characteristics, context, resources, 
goals, and the processes of vocational education, as well as 
about outcomes. And certainly we need to know about correla- 
tions and interactions among the variables. The appeal for 
special attention to particular facets of the vocational ed- 
ucation enterprise is not meant to imply that each component 
stands in isolation from the others. In education, as in the 
economic. world, everything depends on^ everything else. Yet, 
it can still be fruitful to study parts and facets. 



See Daniel L. Stufflebeam, pp. 128-150 in Worthen and 
Sanders, Educational Evaluation: Theory and Practice . 

^^Sce Richard V. Kauffman, A Study of the Educational 
Production Function , Ph.D. thesis. Department of Economics, 
Colorado State University, 1974, pp. 74-31 (reprinted in hard 
copy by University Microfilms, Ann Arbor, 1978)? and Daniel C. 
Rogers and Hirsch S. Ruchlin, Economics and Education; Prin - 
ciples a nd Applications (New York; Free Press, 1971), pp. 
107f, iToff. 
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THE EDUCATIONAL PRODUCTION FUNCTION AS A 
FRAMEWORK FOR EVALUATION 
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C* Vocational Education Outcomes 

The £ocus of this essays however, is on vocational educa- 
tion outcomes r which are broadly defined as the consequences 
of vocational programs. ^ Figure 3 illustrates the range and 
diversity of possible outcomes associated with vocational edu- 
cation programs. All outputs , products, consequences r effects 
results, and impact of vocational programs are recognized as 
outcomes whether they are intended or unintended, positive or 
negative, short-term or long-term, economic or noneconomic^ 
direct or indirect. These outcomes may affect individual stu- 
dents, society as a whole, particular societal entities (such 
as the local school system) , or functional characteristics 
such as the human resource base of the state and local economy 
The outcomes may be seen as resulting from a student's partic- 
, ipation in vocational education or from the existence of on- 
going programs in the community. Outcomes are manifest as 
changes in individual or societal capabilities (including 
knowledge and skills) , attitudes, attributes, status, or cir- 
cumstances. 

Some outcomes, such as the acquisition of occupational 
skills by secondary school students who complete vocational 
programs, are objectively observable and of ten quantifiable. 
Others, such as the development of work-related attitudes or 
leadership capability, may be perceived more subjectively 
while defying straight-forward empirical observation. Ease 
of measurement does not qualify a particular outcome as nec- 
essarily important or valid as a criterion for evaluating 
vocational education, though it certainly may offer practical . 
advantages for such purposes. 

Figure 3 does not purport to identify all the kinds and 
characteristics of educational outcomes. No explicit ::efer- 
ence, for example, is made to physical, esthetic, or moral 
development. The representation is suggestive and impression- 
istic, hence the question marks. Every effect that is 



Definitions of vocational education and vocational 
programs are presented in Section III below. 

14 

A distinction is sometimes made between educational 
outputs and outcomes, the latter implying change while the 
former merely suggests "through-put." No attempt is made in 
this paper to distinguish among such terms as educational pro- 
duct, output, outcome, impact, etc. As indicated in Fig- 
ure 3, outcomes are broadly defined to include all of the 
consequences and results of the education enterprise. 
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AN ILLUSTRAtlON OF SOME OF THE DIVERSE TYPES 
/ AND CHARACTERISTICS OF VOCATIONAL EDUCATION OUTCOMES 



Intanded, Unintetidad 
(Attained Goali, SUto Effects) 




ety 



Benefits, 
Unfavorable 
Consequences 
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significantly linked with a vocational program is regarded as 
a vocatic.ial education outcome* Not all of these consequences, 
however, are readily observable. Professional debate, social 
criticism, and educational research have made the American 
people keenly aware of how much we do not know about the effects 
of schooling. We therefore need to speak of possible outcomes 
of vocational programs without confining the analysis to such 
narrow outcome sets as preannounced goals, presumed benefits / 
or quantitatively measured impact. 

Outcomes typically are multidimensional — e.g., a mix- 
ture of educational and economic conseauences , cognitive and 
affective, individual and societal — and they occur as "joint 
products" in the sense of multiple outputs occurring to- 
gether in a single production process. Thus, just as lumber 
and wood particles are simultaneous outputs of the -.ogging 
industry, so too are occupational skills, job-related atti- 
tudes, and a high school diploma — all "products" resulting 
from the same schooling experience. Which is the "product" 
and which the "byproduct" is a matter of selective per- 
ception. A comprehensive assessment of vocational educa- 
tion outcomes would acknowledge all of the observable con- 
sequences and presumably focus on the ones which are of 
special relevande to the evaluation task at hand. It would 
also recognize the opportunity costs or tradeoffs associated 
with vocational programs, i.e., the alternative educational 
outcomes foregone because the resources are being. utilized 
in vocational programs. 

Differential outcomes may be associated with variations 
in educational treatment (see Section III below) , intensity 
of treatment, i,e., "time on topic"), educational level 
(high school, post- secondary, adult) , student characteristics 
(age, sex, race/ethnicity, socioeconomic backgrovind, physi- 
cal or mental handicap, aptitudes, career interests, etc.) 
and a host* of other variables indicated in Figures 1 and 2 
abov^. This follows from the recognition that variation in 
the "input" factors of the educational production function 
(Figure 2 above) predictably will result in differential 
outcomes. 

Awareness of the broad range, di'^ersity, and multi- 
dimensional nature of possible educational outcomes is im- 
portant to , practitioners, researchers, evaluators, policy 
makers, and the general public. It helps offset the narrow 
perception of equating outcomes sinply with goals or benefits, 
while ignoring "side effects." This expanded view in turn 
triggers a search for evaluative standards beyond preannounced 
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program objectives and can stimulate a re-examination gf 
program implementation strategies and overall policy. 

As an example, when State and local units of government 
first enacted general sales taxes, their single purpose was 
to raise revenue. Attention was focused on the outcome or 
fiscal productivity. Another outcome of the tax, however, 
was its regressive burden: low-income consumers paid higher 
effective tax rates (tax payments as a percentage of income) 
than high-income consumers. While the purpose of the tax 
was not to redistribute income, that was nevertheless an 
outcome. Awareness of this consequence eventually led many 
states to amend the general sales tax by exempting food, 
thereby reducing the regressivity of the tax. 

Similarly, if an important outcome of vocational educa- 
tion is to increase the rate of school completion even 
though the stated objective (or specified evaluation cri- 
terion) is job placement, then knowledge of the school- 
completion outcome can be useful to policy makers as they 
revise legislation and allocate resources for the support of 
various programs. "Side effects" are consequences no less 
than attained objectives. As the environmental, and consumer 
protection fields illustrate, yesterday's side effect may be 
tomorrow's most significant outcome! 

Table I lists 30 questions concerning possible outcomes 
of vocational education. The outcome questions (verbal ex- 
pressions of empirical phenomena) listed here are not intended 



^^In a recent monograph on noneconomic outcomes, the 
U.S. Department of Labor noted the "unfortunate tendency to 
evaluate many employment and training programs in terms of 
narrow economic, cost-benefit analyses alone." The Job Corps 
was cited as "a good example of a program that should not be 
viewed so narrowly — a program in which the noneconomic impacts 
are at least as important and possibly more so, than economic 
benefits such as higher earnings." The report identifies 21 
different noneconomic outcomes in three areas: (1) job re- 
lated; (?) social-attitudinal; and (3) health-related. The 
oi tcomes are discussed in the context of "social development 
objectives . . . rehabilitation goals, and . . . remediation." 
U.S. Department of Labor, The Noneconomic Impacts of the 
Job Corps , R i D Monograph 64, Employment and Training Admin- 
istration (Washington, D.C. : U.S. Government Printing .^Office , 
19 78) pp. iii-iv. More than 200 economic and noneconomic out- 
comes -are identified in Joanne Farley, Vocational Education 
Outcomes: A Thesauru F of Outcome Questions (Columbus: 
National Center for R jearch in Vocational Education, The Ohio 
Jtate University, 1979). 



Table I . 

VOCATIONAL EDUCATION OUTCOME QUESTIONS: ILLUSTRATION ^ 
OF THE RANGE AND DIVERSITY OF POSSIBLE OUTCOMES 


No. Outcome Questions 


Thematic Area 


1. Do students become occupational ly proficient 
and ready for entry^level employment as a 
result of completing a vocational program? 


Skill acquisition 


2. What proportion of former vocational stu- 
dents are employed in training-related jobs 
six months following program completion? 


Employment in 
training-related jobs 


3. Are dropout raLoo lower for vocational stu- 
dents than for conparablo students enrolled 
in general/academic curricula at the sec- 
ondary school level? 


Schooling 


4. Do vocational programs at all levels — high 
school^ postsecondary^ and adult — signifi- 
cantly strengthen the human resource base of 
the local and state econpmy? 

« 


Economic development 


5. One year following program completion^ are 
former vocational students earning higher 
hourly wages than comparable ^ general/academic 
students ^ similar jobs? 


Wage rates - 


6. Are the parents of students who are enrolled 
in vocational programs better satisfied with 
t:he overall high school program than parents 
of students enrolled in general/academic 
curricula? 


Parental 9atisfaction 
with school program 


7. Do vocational students obtain jobs which they 
consider acceptable # whether in training- 
related fields or not^ within a specified 
time period following program con*pletion? 


Employment 


Are vocational programs more effi^ctivo or 
loss offectivo than general /academic cur- 
riiMiKi in onhancinq tho^ moI f-imaao an«i solf- 
confidonc^o of yjtudonts who are preparing for 
employment .* 


Student soir-ima<4t; 

i 



continued 



Table I (Continued) 



VOCATIONAL EDUCATION OUTCOME QUESTIONS: ILLUSTRATION 
OF THE RANGE AND DIVERSITY OP POSSIBLE OUTCOMES 



No. Outcome Questions 


Thematic Area 


9. Do vocational students receive higher employer 
ratings tnan comparajDie general/ acaaeiuic stu^ 
dents following a specified period of employ- 
ment? 


Employer assessment 


10. Do vocational students develop leadership 

ability through their participation in occu- 
patlonai ly— rei|iteci youun programs sucn as 
Distributive Education Clubs of America 
(DECA) , Future Homemakers of America (FHA) , 
Future Partners of America (FPA) , and Voca- 
tional Industrial Clubs of America (VICA)? 


Leadership develop- 
ment 


11. Are students who are currently enrolled in 
vocational programs better satisfied with 
their overall school experience than compar- 
able students 'enrolled in general/academic 
curricula? 


Student satisfaction 
with schooling 


* * 
12. Do graduates of vocational programs express 
more, less, or the same job satisfaction as 
comparable graduates of general/academic 
programs holding similar jobs? 


Job satisfaction 


13. Does participation in high school vocational 
programs reduce the incidence of school 
vandalism and other antisocial behavior on 
the part of students? 


Antisocial behavior 


14. Does vocational education (high school^ post- 
secondary, adult) provide its graduates with 
a general set of skills and basic work dis- 
ciplines that significantly increase their 
chances for successful employment? 


Employ ability 
development 



continuea-' » 
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Table I (Continued) 

VOCATIONAL EDUCATION OUTCOME QUESTIONS: TLLUSTRATin'J 
OF THE RANGE AND DIVERSITY OF POSSIBLE OUTCOMES 


No. Outcome Questions 


Thematic Area 


15. By completing a vocational program, do stu- 
dents acquire job-seeking skills such as how 
to write a resume, how to participate in an 
interview, and knowing where to look for job 
oppo rt unities? 


Job-search skills 


16. Do students enrolled in vocational programs 
learn the basic cognitive skills of speaking, 
reading, writing, an^ numerical 'calculation 
as effectively as comparable students in 
academic/general curricula? 


Educational achieve*- 
ment 


17. Does participation in a vocational program 
give students a realistic understanding of 
what employers will expect of them on the job? 


World-of-work 
understanding 


18. Is the occupational mobility ->f vocational 
r>tudents limited both horizontally and 
vertically by the content of the programs 
(e.g., training in narrow skill areas or in 
occupations with diminishing job opportuni- 
ties)? /* 


Occupational mobility 


!')• Doos participation in a high school vocational 
program inhibit or enhance a student's oppor- 
tunitic^s for a college education? 


College education 


20. Do studonts who complete vocational programs 
have a more realistic understanding of our 
society and its economic system than compar- 
.iblo studont?5 who complete goneral/acatiomic 
programs? 


Socioeconomic 
understandi nq 


Jl. Are thi> roturrus from individual and social 
irm^*^^mtMlt s in vo<rAtional oducation lower, 
hii?h#»r, or tfio samp an ro^^Jrns from 
i nv♦»s^mon^ ma<i*> in gotu t :j 1 /acrademio pr(->- 
grams bt^h^w f.hv haocalaurrato lovfl? 
• 


Kronomic returns 
to schooling 

! 

1 

_ 1 



Table I (Continued) 

VOCATIONAL EDUCATION OUTCOME Ql^lSTIONiS: ILLUSTRATION 
OF THE RANGE AND DIVERSITY OF POSSIBLE OUTCOMES 


No. Outcum'^j puestions 


Thematic Area 


22. . Does participation in vocational programs 
on the high school level contribute measur- 
ably to the mortal development of students? 


Moral development 


23. Through their participation in vocatiqnal 

education progreuns (e.g.# cooperative educa- 
tion, placement in part-time jobs), do dis- 
advantaged minority youth earn more income 
while still in school than comparable students 
in general/academic programs? 


Student earnings 

• 


24. Do high school, postsecondary, and adult 
vocational programs increase occupational 
awareness, training opportunities, and job 
access for women in both traditional and 
non traditional areas? 


Employment opportun- 
ities for women 


25. Does favorable community reaction to ongoing 
vocational programs result in general expres- 
sions of satisfaction and financial support 
for the entire local school system? 


Community support 
for education 


^ry • iJO oCUQeni-S WnO lla,VtT ^CIX. U-LC A-^€« ucva vws#a\-AW«*aA 

progreuns on the high school level have more 
clearly defined occupational goals than those 
expressed by nonvocat lonal students who enroll 
for. postsecondary uoation? 


Career planninci 


27. Do high nchool home econcMitics courses sig- 
nificantly incroaso the consumer skills of 
students who have taken thorn? 


consumer skills 



continued 
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, Table I (Continued) 

• • ; 

VOCATI(»(AL EDUCATION OUTCOME QUESTIONS: ILLUSTRATION 
OP THE RANGE AND DIVERSITY OF POSSIBI£ OUTCOMES 


No* Outcome Ouestions 


Th'4»natic Area 


28. Do certain admissions and student assignment 
practices followed in vbcational education 
tend to reduce or to reinforce occupational 
discrimination on the basis of race, class » 
or sex? 


Occupational dis- 
crimination- 


29. Is job«*search time lower for students who 

participated in vocational programs, than for 
comparable workers who did not? ^ 


Unemployment 


30. How do the annual earnings of vocational grad- 
uates compare with those of general/academic 
graduates, both in the short term and in the 
long run? 


Earnings 



(continued from page 14) . 

to imply anything about what we actually know, believe, or 
suspect concerning the consequences of vocational programs. 
The questions simply illustrate the broad and diverse 
range of possible outcomes. 

It is interesting that while some of the outcome 
questions relate to employment and earnings » other focus 
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on such topics as schooling, antisocial behavior » economic 
development, parental attitudes, and the like. Some of the 
questions imply positive consequences for vocational pro- 
grams, while others may suggest negative or neutral effects. 
What is quite clear is that even a limited array of voca- 
tional education outcomes, as we have illustrated here, ex- 
tends well- beyond traditionally announced program object-ives, 
labor market benefits to individual students, and alleged - 
harmful or beneficial effects on educational development. 

Several of the outcome questions in Table I have bf|n 
addressed in empirical research and evaluation studies. 
Undocumented assertions have been made regarding the facts 
alluded to ih other questions. And some of the questions 
have simply been posed without conjecture as to the answers. 
At this point in the discussion, in keeping with our intent 
to expand awareness of the range and diversity of outcomes, 
we simply wish to regard these questions as illustrative 
verbal expressions of. possible results: waves and rip^i^es 
that vocational programs may or may not be generating. 



D. Outcomes and Outcome Statements 

Figure 4 outlines a conceptual framework for evaluating 
vocational education with respect to outcomes. The outcomes 
themselves (1) are empirical phenomena: real-world conse- 
quences of educational programs. Outcome statements (2) 
are verbal expressions of these cause-and-ef feet phenomena. 
Educational outcome statements can take the form of declar- 
ative sentences or hypotheses: "Completers of high school 
vocational prdgreuns have lower rates of unemployment six 
months after graduation than comparable completers of aca- 
demic/general curricula." Or they can be phrased as ques- 
tions (as in Table I) : "Do completers of high school 
vocational programs have lower rates of unemployment six 
months after graduation than comparable completers of 
academic/general curricula?" An outcome statement describes 
what happens to 



•^^For descriptive and evaluative comments on these 
studies see Holland, Evaluative Bibliography . 

^^See Farley, Thesaurus for a categorized listing of 
228 vocational education outcome questions along with an 
analysis of the structure and content of outcome statements. 
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EDUCATION OUTCOMES EVALUATION FRANIIWORK 

(1) EDUCATIONAL OUTCOMES 

o 

(2) OUTCOME STATEMENTS/HYPOTHESES/QUESTIONS 



(3) OUTCOME MEASURES (DATA, EMPIRICAL INDICATORS) 

o 

(4) EVALUATION'STANDARDS (RELATIVE, ABSOLUTE) ^ ^ ^ 0 

o 

(6) ANALYSIS, INTERPRETATION, AND EVALUATION 

o 

(6) FINDINGS CONCERNING THE CONSEQUENCES OF PARTICULAR PROGRAMS 

o 

(7) APPLICATIONS OF FINDINGS 

{7'A) • POLICY (GOALS, RESOURCES, RESOURCE ALLOCATION) 

(7 B) • PROGRAM DESIGN 

(7.C) • PROGRAM MANAGEMENT 

(7D) • ACCOUNTABILITY 



Figure . 4 
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whom and how, and in some cases also indicates when, where, 
and why. As a minimum, a meaningful outcome statem<?nt 
clearly identifies (a) an outcome, (b) the person or thing 
affected, and (c) the educational treatment. Often the 
statement (qr outcome question) will also identify (d) the 
agency through which the treatment is applied, (e)* a rationale 
for the posited outcome, and (f) a time framework. Expressing 
an educational outcome in precise verbal form affords two 
important benefits: cls^rity of understanding and clarity of 
communication, hs Roget of Thesaurus fame and others have 
pointed out, words are not only a medium of communication, 
but equally important, they are instruments of thought. 

In Table I, questions are used rather than statements 
or hypotheses. This is to avoid giving an impression that 
we are asserting beliefs or . reporting .documented facts. 

Outcome measures (point 3 in Figure 4) are empirical 
indicators of the educational outcomes. These indicators 
may be quantitative (differences in amount, expressed 
numerically/statistically) or qualitative (differences in 
kind, described verbally) The outcome measures are ob- 
servable indicators of the changed conditions or circum- 
stances resulting from an educational program. These 
changes may be observed directly or indirectly. Outcome 
measures require empirical information such as employment 
and wage data, and standards or norms for evaluating the 
data. It is at point 4 in the framework, therefore, that 
positive analysis of educational outcomes (observation and 
descripiton) merges with normative analysis (appraisal 
of data) to produce the judgment of worth that constitutes 
educational evaluation (5) . 

The findings about program consequences (6) that result 
from evaluation research become available for a variety of 
applications (7) including policy making, procrram improvement, 
accountability, and decisions to terminate, expand, or 
otherwise modify the education enterprise. Returning to 



The author is indebted to Herbert A. Simon, winner of 
the 1978 Nobel Prize in Economics, for Roget 's observation: 
*'The use of language is not confined to its being the medium 
through which we communicate our ideas to one another; it 
fulfills a no less important function as an instrument of 
thought; not beinq merely its vehicle, but giving it wings 
for flight." Herbert A. Simon and Allen Newell, "Models: 
Their Uses and Limitations" in Leonard D. White (editor) 
The State of the Social Sciences (Chicago: University of 
Chicago Press, 1955), p. 66. 
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4, we want to emphasize the point that introducing 
value criteria (standards of goodness or badness) into the 
research process is prerequisite for making evaluative 
judgments. Whether a particular outcome such as the place* 
rent of 65% of all vocational graduates in training-related 
jobs is considered good, bad, or indifferent depends oh 
the evaluative standards applied. Knowledge of facts alone 
may be sufficient for research, but in the absence of a 
standard for 'comparison, there can be no meaningful 
evaluation. 

Standards or norms for evaluating observed outcomes can 
thus be regarded as the link between positive outcomes 
analysis and evaluation of vocational education in terms of 
outcomes, which by definition is a normative undertaking. 
Where these standards come from eind how they are used are 
therefore, fundamental issues to be considered in this paper.' 



III. SOME KEY ISSUES IN VOCATIONAL EDUCATION OUTCOMES EVALUATION 



As noted by the authors of a Rand Corporation report on 
the findings of educational research, "Education is an ex- 
tremely complex and subtle phenomenon." Moreover, educa- 
tional research is subject to serious limitations of data, 
measurement instruments, and experimental controls, includ- 
ing the influence of factors outside the schools ("perhaps 
strong enough to 'swamp* the effects of variations in educa- 
tional practices"). The upshot is that "research has found 



Jerome Moss, Jr., The Evaluation of Occupational 
Education Programs , Technical Report, Research Coordination 
Unit m Occupational Education, University of Minnesota, 
September 1968, p. 5. 

20 

As suggested in Section I, outcomes analysis addresses 
two types of questions: What are f.he actual consequences of 
vocational education? and What are che desirable consequences 
of vocational education? The first is an empirical matter*^ 
of-fact question triggering a search for information about 
what is. The second is a normative question leading to a 
search for values indicating what ought to be . To the ex- 
tent that value judgments are influenced by evidence and 
reason, the search for values and facts will be intertwined. 
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nothing that consistently and unambiguously makes a diffet* 
ence in student outcomes." 



While the Rand assessment dealt with general education 
(and in the context of the federal government's role in 
school finance) , the conclusions are not unlike those re- 
ported above for vocational education. Indeed, vocational 
education research and evaluation faces all of. the general 
problems confronting education plus some special difficul- 
ties of its own, to which attention is now turned. 



A. What is Vo9ational Education ? 

The study of educational outcomes, as illustrated in 
Figures 1 and 2 above, investigates the consequences of 
distinctive educational treatments or interventions. What . 
is the nature of the treatment carrying the label "voca- 
tional education"? 

According to the Education' Amendments of 1976 (Sec. 195) , 
the term vocational education means "organized educational 
programs which are directly related to the preparation of 
individuals for paid or unpaid employment, or for additional 
preparation for a career requiring other than a baccalaureate 
or advanced degree." In further defining an organized edu- 
cation program, the law refers to "instruction related to 
the occupation or occupations for which students are in 
training or instruction necessary for students to benefit 
from such . training." 

In a generic sense, vocational education can be defined 
as any program of education or training that enhances one*s 
capability to pursue a vocation, i»e., a calling or occupa- 
tion or career. Definf^d more narrowly for programmatic 



Harvey A. Averch and others. How Effective Is School - 
ing: A Critical RoView and Synthesis of Research Findings , 
Rand Corporation, Santa Monica, Calif., March 1972. See 
pp. ixf , 154, 160. The authors emphasize that they are 
"not suggesting that nothing makes a difference, or that 
nothing 'works' y the problem is that research findings 
often conflict." In a conversation with the author of this 
paper, Michael Scriven pointed out, however, that cognitive 
outcomes do consistently show up. It is when educat^ors 
"slide into grandiosity" searching for impact on life style, 
income distribution, and similar variables that planned var- 
iations in education seem not to work. 
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purposes, vocational education refers to occupational pro- 
grams which are included in state plans developed by the re* 
spective State Boards for Vocational Education. A similar 
administrative definition limits vocational education to 
programs supported by Federal funds. In practice, "the 
vocational education program has become so broad and diverse 
in its response to manpower demands and the needs of all 
ages and ethnic groups that is almost impossible to de- 
scribe in any definitive way." 

One may question whether it makes sense to speak of 
"the" vocational education program, not only because of the 
manpower demands and needs alluded to, but for at. least two 
additional reasons. It has frequently been observed that 
there is no national vocational education program in the 
United States. We have 50 different state programs eadh 
with its own suppose'd needs, resources, goals, priorities, 
standards, definitions, data requirements, and delivery 
systems. In light of these differences, are there enough 
commonalities among state programs to discern a vocational 
education program that is reasonably "typical" for the 
nation? The answer would seem to be "Yes." Despite some 



Vocational education is defined in Encyclopedia 
Brittanica as ". . . instruction that is intended to fit 
persons for industrial or commercial occupations" (Vol. 23, 
p. 95) . The Dictionary of Education (edited by Carter V. Good) 
defines vocational education as "a program of edvication below 
college grade organized to prepare the learner for entrance 
into a particular chosen vocation , or to upgrade em'-loyed work- 
ers; . . . training or retraining ... is given in schools or 
classes under public supervision and control or under contract 
with a state board or local education agency" (p. 645) • In 
Ohio State University's Proposal for a National Center for Re- 
search in Vocational Education, a distinction is noted between 
"Vocational Education (capitalized) , the program supported 
under federal legislation; and vocational education in a gen- 
eric sense . . . [which] embraces all activities, public and 
private, which are designed to contribute directly to occupa- 
tional proficiency" (Vol. I, pp. If). Examples of such activ- 
ities cited in the proposal include military training. Compre- 
hensive Employment and Training Act (CETA) programs, public and 
private school vocational programs, and apprenticeships. 



Mary L. Ellis, A Report to the Nation on Vocational 
Education (College Park, Maryland: Ellis Associates, 1975) , 
p. 2. 
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Interstate differences in Student Characteristics (n. in 
Figure 2 above) and Context (n ) it seems reasonable to this 
writer that nationally shared Goals (n.) and the existence 
of national markets for Resources (n^) and. Processes (n ) 
would combine to establish vocational education programs in 
the fifty states that are in essence quite similar. If this 
is in fact true^ some important outcomes should be observable 
as a national pattern. 

The other respect in which diversity might undermine 
the plausibility of treating vocational educeUBion as an 
entity relates to differences among particular occupational 
programs. Does' bistributive Education, for example, have 
less in common with Vocational Agriculture than it does with 
the academic/general school curriculum? Are inter-program 
differences in goals, methods, personnel, and philospphy so 
vast that the occupational curricula share little other than 
the "vocational" label? Is it true, as some have claimed, 
that "There are no vocational educators,'* only Vo Ag, Home Ec, 
and T & I educators"? Operating on the premise that the 
more specialized and unique a program is, the easier it is 
to demonstrate differential outcomes, a case can be made for 
evaluating highly specific occupational curricula (such as 
Air Conditioning, Automotive Ser 'ices, Diesel Mechanics) 
rather than clusters (such as Trade and Industrial Occupa- 
tions) . However promising this "component-program" 
approach po evaluation might appear, it would nevertheless 
pose methodological problems of its own when trying to 
aggregate results for the purpose of producing generaliza- 
tions about "the outcomes of vocational education." 

B. Vocational Programs as Educational Treatments 

Answering the question. What is vocational education? 
is more than an "academic" exercise. If evaluation entails 
both description and appraisal as suggested in Section II 
above, a necessary first step for evaluation is to identify 
the entity to be described. What are the essential features 
oC the distinctive educational treatment or intervention be- 
ing evaluated? Figure 5 depicts the relationship between 
"input" variables and educational outcomes in a manner that 
differs somewhat from the production function of Figure 2. 
Intervention (or program treatment) is defined in Figure 5 
to include Resources (n^) , Goals (n.) , and Processes (n^) ; 
while Students (n j) and Context (nj? are ucpicted separately. 
This formulation has the dual advantage of isolating the two 
"input" variables that are less amenable to variation while 
also highlighting the fact that outcomes iTtay be observed 
with respect to either (a) individual students or (b) social 
context. The linkage on the left side of the schematic in- 
dicates continuous mutual interaction between Students and- 

« 

26 



31 



DETERMINANTS OF VOCATIONAL EDUCATION OUTCOMES 




VOCATIONAL EDUCATION OUTCOMES 

• FOFt i^MDlVlDUAL STUDENTS > 

• FOR SOCIETY 



*lntarvention or educational treatment is narrowly defined here to encompan 
Rotourcei (n,). Goals (n4), and Processes (ng). (See Figure 2 above.) 

**Observal *« effacts of distinctive educational treatments (subject to the 
influence oi extraneous forces and error factors). 



Figure 5 0 
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Context. Using this framework, we can build on the dis- 
cus n of program definition in the preceding section. 

The focus there was on interstate differences in voca- 
tional education programs and possibilities of disaggrega- 
tion based on occupational clusters or sbecific program 
curricula. We now want to approach the unlt-o^f -analysis 
issue from the broad perspective of the tota^/educational 
program. The question may be posed: Is it appropriate for 
evaluation purposes to consider vocational education » however 
that entity may be defined, as a treatment distinct 
from the total educational program (especially the occupa- 
tional skills component)? 

In discussing 'the issue of criterion variables for 
vocational education, . one observer points to the "separation 
of vocational and general education** as a major source of 
difficulty. •*As long as vocational education claims to be 
the sole input for the knowledge, skill?, and attitudes re- 
quired for occupational competence,** he argues, "it will be 
vocational education, not the total school program, that 
takes the brunt of criticism that schools are not accomplish- 
ing satisfactorily their functions of occupational ... 
preparation." In other words, if outcome criteria such as 
occupational proficiency, employability, actual employment, 
work adjustment, "satisfactoriness" to employers, and a 
successful transition from school to work are to be used for 
evaluating program performance, should the treatment that is 
being evaluated be (a) the total school program, or (b) the 
subset of courses and experiences labeled vocational educa-' 
tion? 

» 

Whether vocational education does indeed claim to be 
the sole source of the components of occvipational coropetency- 
or has such a function ascribed to it — is a moot point. The 
practical question for vocational education evaluation is 
whether (a) or (b) is regarded as the treatment. If the 
former, then "evaluation of the school's occupational educa- 
tion function (for all of the school's students] becomes a 
complicated ana methodologically sophisticated affair . . . 



Warmbrod, "Evaluation Research in Vocational 
Education," pp. ^f. 



28 

H6 



something of the magnitude of the Coleman Report."^' It might 
be conjectured that this strategy would also tend to remove 
the focused heat of educational accountability fforo vocation- 
al education. 

Yet a legitimate question remains. So long as schools 
offer a course or set of courses and experiences under the 
rubric of vocational education — and with earmarked funding — 
there will be a reasonable expectation for an educational 
quid pro quo. What comes out of the vocational education 
process in response to the specialized resources that go 
into it? Perhaps the outcomes that can reasonably be ex- 
pected will be much more limited than occupational competence, 
employability, employment, successful work adjustment, and 
the like. Some differential outcomes, however, should pre- 
sumably be identifiable as resulting from distinctive goals., 
processes, and resources. 



C. Evaluative Criteria and Standards, Experimental Controls ^ 
cmd Comparison Groups 

According to one of the definitions listed in Section I, 
evaluation is the process of comparing performance data with 
clearly specified objectives. This conception of evaluation 
is associated with the currently popular view that the sole 
or dominant purpose of evaluation is program improvement: 
"Evaluation is not to prove, but to improve."^ The approach 
helps explain why many people equate a study of vocational 
education outcomes with an investigation of goals and the 
extent to which they have or have not been achieved. Clearly, 



Warmbrod argues (on p. 9 of his Beacon article): 
"When vocational education is viewed as a function of the 
total school progr£un, some important issues relevant to the 
design and conduct of evaluation studies come into focus." 
Fiist, the tarcfet population becomes all of the school's 
students, not just vocational students. Second, the treat- 
ments to be evaluated are various mixes of vocational and 
other courses. And third, "an evaluation of the school's 
occupational education goal becomes a part of the evaluation 
of all the goals and functions of the schools." 

26 

Floyd L. McKinney, Program Evaluation in Vocational 
Educatipn: A Review (Columbus : The ERIC Clearinghouse on 
Career Education, Center for Vocational Education, Ohio 
State University, 1977) p. 2. 
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however, the observable effects of vocational programs 
may differ from the preannounced intent of such programs not 
only in terms of discrepancies between stated outcome goals 
and actual achievement, but also with regard to such side 
effects as (a) benefits not previously identified aS goals 
and (b) other consequences that might be considered neutral 
or negative. In short, evaluation generically defined can 
serve other functions besides program improvement; and 
evaluative criteria can be derived from sources other than 
preannounced objectives. 

It was noted in the Introduction that one function of re- 
search is to describe the consequences of given actions, such 
as the observable outcomes of particular educational programs* 
Positive outcomes analysis, as defined above, addresses the 
full range of educational outcomes, not just traditional ob- 
jectives and presumed benefits. This breadth of scope can 
produce insights which in turn may generate new goals and 
priorities. In this way, knowledge of what is can contribute 
to informed judgments of what ought to be . iT a stated cb- 
jective of vocational education is preparation for employment 
in a training-related occupation but an observed and perhaps 
unanticipated outcome is employability in' a much broader 
range of occupations, with corresponding labor market bene- 
fits for individual students, then the latter goal might 
appropriately be added to or substituted for the narrower one. 

As we earlier emphasized, in tioving from positive 
analysis to evaluation, there is a need for evaluative 
standards. Some of the standards may be goal-based; but 
lacking a comprehensive, omniscient statement of program ob- 
jectives, where does the full set of evaluation standards 
come from? One source is professional judgment as to what 



It has been noted that an area "not well addressed 
in writings on the evaluation process is the establishment of 
standards and the relative utility of various types of 
standards" ( Quentin D. Clarkson and others, "A System for 
Establishing Evaluation Standards," CEDaR Quarterly, Vol. 10, 
No. 1, spring 1977, pp. 17-19) . In their proposed classifica- 
tion system, the authors distinguish four types of relative 
standards (Normative, Predicted, Cont;:ol, Trend) and two 
types of absolute criteria (Arbitrary or intuitive, and Value- 
based) . Moss and Stromsdotf er (."Evaluating Vocational and 
Technical Education Programs," p. 224) have observed that "the 
problem of evaluating vocational education is confounded by 
the fact that its* objectives and outputs are multidimensional 
. . . [and] successful development, weighting, and estimation 
of performance^ indexes . . . has never been achieved." 
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constitutes reasonable and appropriate performance. Agree- 
ment might be reached (by a panel of evaluators or their 
sponsors?) that an unemployment rate under 10% constitutes 
an acceptable programmatic outcome. Or a flexible norm could 
be established for vocational graduates pegged at say 75% of 
or 110% of the current unemployment rate for all labor force 
l^articipants in the same age group. . ^ , 

Evaluative standards can be derived from observing the 
outcomes of similar program activities, e.g.., job-search 
time required for CETA graduates to find unsubsidized jobs, 
compared with job-search time for vocational school gradu- 
ates, or differences in outcomes resulting from the same 
program observed over a period of years. 

And, in certain instances, verbal absolutes such as 
"satisfactory,- "high.," "low," "good," "bad," can be posited 
as non^mpirical standards. These seem less likely to prove 
useful /and credible than criteria based on empirical ob- 
servations (whether quantitative or qualitative) . 

/Perhaps the most promising source of evaluation stand- 
ards is needs assessment, which recognizes that someone * s 
desiderata or "justified goals" — though not necessarily the 
sponsor's or producer's — are appropriate criteria for eval- 
uating educational programs. 

I • 

J Once again, definitional problems intrude on the issue 
of evaluative criteria. Who are "vocational . students, " and 
who are the "nonvocstional students" with whom status or 
perfcjrmance might usefully be compared? A host of related 
issues arise, including the use of experimental controls and 
comparison groups. Definitions aside, is it appropriate to 
compare employment status and earnings of former vocational 
students with nonvocational (i.e., academic and general) 
students? Should comparisons of labor market status and 
performance (e.g., finding employment in occupations related 
to training) only be made among specific occupational 
curricula within vocational education, or can such comparisons 
be made legitimately, between the totality of vocational stu- 
dents on one hand versus all other students? If the latter, 
which ^^articular entry-level jobs are "training-related" for 
graduates of the high school academic and general curricula? 

Consider the mandate under the Education Amendments of 
19.76 to evaluate vocational programs "according to the extent 
to which program completers and leavers (a) find employment 



Scriven, "Evaluation Perspectives and Procedures," p. 51. 
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in occupations related to their training, and (b) are con- 
sidered by their employers to be well-trained and prepared 
for employment." What are the procedural options for carry- 
ing out the evaluation? As an illustration, without exten- 
sive discussion, suppose the strategy is limited to identify- 
ing the former vocational students who have found training- 
related ^obs (however defined) and then surveying their em- 
ployers with given instruments. Assume that 60% of "program 
completers and leavers" found jobs in occupations related to 
their training and within this group, 80% were rated highly 
satisfactory with respect to their training and preparation 
fo^ employment. How are the data to be evaluated? As noted 
in Section II, whether outcomes are comiclered good or bad 
depends on tho comparison standards ep'.ployed. 2 9 

It was suggested above that in the absence of clearly 
defined performance objectives or predetermined evaluation 
standards, it will be necessary to derive or develop such 
standards. One approach is to compare the status or per- 
formance of vocational students with another group, similar 
in all important respects — age, sex, mental ability, socio- 
economic background, etc. — except that the vocational students 
received a distinctive educational treatment that was not 
provided to the comparison group. In the case of secondary 
school students, the comparison might be made for vocational 
students versus nonvocational (i.e., academic plus general) 
who are similar in all characteristics except the treatment 
variable (participation in a vocational program) . Given such 
a comparison, is it reasonable to hypothesize significant 
differences in the outcome variable for the respective groups? 

The general rationale for the hypothesis is that sub- 
stantially different educational treatments can be expected 
to yield significantly different outcomes. Whether the dif- 
ferences between "vocational education" and the "regular 
treatment" can be expected to result in enployability 
differences is a more specific and problematic guestion. Of 
the two outcome variables mentioned in the Education Amend- 
ments of 1976, placement in a training-related job 

• 2 9 

It miqht be argued that interpretation of the facts and 
evaluations can be left to others — managers, policy makers, 
interested parties — who can furnish their own judgmental 
criteria. The question of who does the evaluating, however, 
does not diminish the importance of how it is done. Procedures 
and value premises will be of concern to all groups having a 
stake in the evaluation, and whoever supplies the values will 
presumably be hold accountable. Dogmatic standards and vote- 
counting may seem expedient, but that leaves important value 
questions unanswered. 
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is a highly questionablft evaluation criterion for 

at least three reasons s (a) '^relatedness** is ambiguous; 

(b) vocational students who were enrolled in a particular 
curriculum may not. desire employment in a related occupation 
(either because their preferred option was not oflered and 
they simply took something that was available, or because 
the option they followed provided a useful opportunity to 
explore and reject a particular occupational tif^lf.;} 

(c) there are no appropriate comparison groups fox whom 
differentiated fields of training can be identified* 

On the other hand, the satisfactoriness-to-employers 
criterion.could.be applied in the following way. One could 
enumerate the former nonvocational students in a given school 
class who found jobs (irrespective of occupational fields) , ^ 
draw a sample matched to the vocational students according to 
characteristics such as mental ability and socioeconomic 
background, and then compare employer assessments of voca- 
tional and nonvocational students relative to the extent to 
which they are considered "well-trained and prepared for 
employment." « . 

If the intent were to test the efficacy of the disk- • 
tinctive vocational education treatment with respect to~^ 
employability (as measured by employer assessment of training 
and preparation) , it would seem plausible to expect lower 
rates of unemployment for students exposed to an employment- 
related vocational education treatment as compared with 
students who were not.^° If no significant differenpes occur 



Such expectations are plausible for the same reason 
one would expect a student who completed three years of 
Freneh- to enjoy an- -adv antage practice of that IsTiguage 

over a comparable student who had spent three years taking 
history instead. A separate question relates to the amount 
of the differential in unemployment rates. . Is achieving the 
differential worth whatever extra costs, if any, have been 
incurred in providing the vocational education program? 
Efficiency issues involving calculations of marginal benefits 
and marginal costs are not addressed in this paper, for two 
reasons.^ First, until we know more about gross outcome 
variables, it seems pointless if not pernicious to go through 
the exercise of manipulating the finely tuned pecuniary values 
needed to generate benefit-cost estimates. Second, there is 
so much methodological -discretion inherent in performing 
benefit-cost, rate-of -return, and cost-effectiveness analyses 
that the findings may reflect the tastes and style of the in- 
vestigator more faithfully than the nature of the empirical 
evidence. 
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between the matched groups, or if the regular group performed 
better than the treatment group, this certainly would not 
demonstrate that vocational education was an abysmal failure. 
It would suggest, however, that success criteria might be 
sought elsewhere; perhaps vocational programs do not enhance 
employability differentially but have other worthwhile outi^omes. 
Alternative tests can presumably be designed to measure these out 
comes, thereby helping to disclose the real nature of the voca- 
tional education outcomes function, i.e., the matrix of conse- 
quehces associated with thia distinctive educational treatment* 



D. Nature and Quality of Outcome Da^ta 

' Numerous studies have commented on the availability and 
interpretability of vocational education data. Evaluators 
have lamented the lack of national data, accurate data, 
comparable data, "hard" data, cost data, and impact data.-*-^ 
Our own review of empirical outcome studies and related 
inquiries concerning the credibility and persuasiveness of 
outcome data suggest the need for a profile of data types 
classified on the basis of the nature of the data and how 
they are generated. Table II distinguishes five types of 
data used in describing and appraising the consequences of 
vocational education. 



^^Among the data-related problems identified by McKinney, 
~UraV7 "and "Abram " ( int6YPYei:iiic|j Oufcc5mg~ measures ; p . IT , are 
those of "definitions, communication, and problems of handling 
data." In the same report (Ch. Ill), Donald W. Drewed comments 
on data format and other issues associated with setting up 
the proposed Vocational Education Data System (VEDS) . For 
additional perspective on data problems, see Arthur M. Lee 
"Project Baseline: Historiographic Foundati'ons for Vocational 
Education Statistics, " Educational Researcher , Vol. 6, No. 7 
(July-August 1977) , pp. 3-9. As indicated in footnote 2 
above, two state-of the-art papers dealing with vocational 
education data are scheduled for publication by The 
National Center in 1979. See Borland, Evaluative Bibliography 
for conntients on data used in some recent studies of <iVocationai 
education outcomes. 
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Table II 

TYPES OF DATA USED IN VOCATIONAL EDUCATION OUTCOMES EVALUATION 



Type of Data 


Nature of > 
Observation 


Method of Observation 


Bxttiples 


Status 

(Nonmone tary ) 


Objective 


Observation and reporting of 
circiuastance by a second par«* 
ty or directly by respondent. 


Labor force status of former voca«» 
tional students* 

School enrollment status of 8tu«» 
dents currently or formerly 
enrolled in vocational programs # ^ 


Status 

(Monetary) 


Objective 


Observation by a second party 
of circiamstances reportable 
in n^ney terms* 


Hourly wages of fonaer vocational 

and nonvocational students* 
Annual earnings of former students* 
Cost and benefits of vocational 
programs * 


jPersonal Judq- 
1 ment 


Subjective 


Expressed directly by 
respondent • 


Student opinions or perceptions 
concerning quality and relevance 
of vocational programs* 

Employer assessment of the extent 
to which students are well trained 
and prepared for ^ployment* 


Ascriptive 


Objective 


Characterization by a second 
party based on records^ test 
results^ and/or observation 
and professional assessment* 


Work-*related attitudes and attri** 

butes of students* 
Vocational interests and aspira^ 

tions of students* 


Performance 


Objective 


Written or other tests of cog-» 
nitive and psychomotor know-- 
ledge and skills* 


Score on skill proficiency test 

(e*g* 9 welding) * 
Score on reading/math achiev^ent 

test* 



The literature review to which we allude disclosed 
heavier use of Status (Nonmonetary) and Personal Judgment 
data than the other types. In part, this reflects concern 
with the outcomes specified as evaluation criteria in the 
Education Amendments of 1976, namely "the extent to which* 
vocational program completers and leavers . . . find em- 
ployment in occupations related to their training, and . ... 
are considered by their employers to be well-trained and 
prepared for employment" (Sec. 112). Certain kinds of data 
are clearly more appropriate than other types as empirical 
indicators of particular outcomes (see Figure 4 above) . in 
fact, outcome hypotheses can be formulated so explicitly 
that given types of data and particular evaluation standards 
are specified for use in performing the evaluation task. 

A.,. , Additional factors influencing the kinds of outcome 
data used m evaluation include ease, timing, and costs of 
collection; privacy considerations; credibility; comparability 
with other available data; amenability to statistical treatment 
and analysis; potential impact; and the ingenuity and skill 
of evaluation designers and data gatherers. Inevitably, there 
are tradeoffs between such considerations as the ease and 
economy of collecting softer data of the Personal Judgment 
type and generating "hard" Monetary data, or Performance data, 
which might have higher impact on certain audiences. 



IV. SUMMARY AND RECOMMENDATIOIIS 



As an "instrumental" contribution to answering the 
research question "What particular outcomes are app?opriate 
^rL«?« ^f^?^^teria to evaluate vocational education pro- 
grams? this paper has attempted to identify and clarify 
some key concepts and issues related to educational outcomes. 
Three additional products were completed during the first 
year of the National Center's study of vocational education 
outcomes and are recommended as companion references: (1) 
Vocational Education Outcomes; An Evaluative Bibliography o f 
Empirical Studies ; (2) Vocational Education Outcomes : A 
Thesaurus of Questions ; (3) Vocational Education Outcomes : 
Annotated Bibliography of Related Literature . It is hoped 
that these products will help stimulate further conceptual and 
empirical work in the field and prove useful as points of 
reference for future commentary and reporting. 
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A, Summary 



Vocational education outcomes were defined In this 
paper as "the consequences of vocatlonar programs. ** Figure 3 
depicted the range and diversity of possible outcomes asso- 
ciated with high school, post-secondary, and adult vocational 
education programs. It was emphasized that educational out- 
comes are not limited simply to the goals, objectives, and 
benefits of vocational programs but also Include unintended 
consequences ("side effects") which may be evaluated In 
positive or negative terms. Expanded awareness of the range 
and diversity of possible outcomes can help offset narrrow 
perceptions that. Inhibit research, evaluation, program Im- 
plementation, and policy making. Table I listed 30 vocation- 
al education outcome questions. Illustrating such diverse 
themes as the acquisition of knowledge and occupational 
skills, economic development, employment, ^arnlngs, leader- 
ship development, student and parental satisfaction with 
schooling, educational and occupational discrimination, job 
satisfaction, personal self-image, occupational mobility, 
work-related attitudcrs, career planning, job-search skills, 
employment opportunities for women, and economic returns to 
schooling. Some of ^he outcomes identified in these ques- 
tions apply to individual students while others are more 
broadly societal in nature. 

An analytical framework for evaluation was suggested in 
Figure 2 ba&ed on an educational production function and sim- 
ilar to the familiar CIPP model (Context, Input, Process, and 
Product) . Figure 1 identified six components of the education 
enterprise: Student Characteristics, Context, Resources, Goals 
Processes, and Outcomes. It was suggested that vocational 
education can be evaluated not only on the basis of what comes 
out of the system— educational products, outputs, effects, 
impact — but also in terms of what goes into the enterprise and 
how it functions . While outcomes are of great Importance, it 
cannot be claimed that this is the only category of criteria 
relevant for appraising the merit of vocational education. 
And within the outcomes category, one must resist the temp- 
tation to look only at one or two particular outcomes while 
ignoring all the other consequences of vocational programs. 
In order to evaluate vocational education in a balanced and- 
meaningful way, evaluators should avoid putting all of their 
eggs in one narrow basket. Figures 1 and 2 were intended to 
provide perspective and focus attention on outcomes. 

Figure 4 dlst^ingulshed among educational outcomes 
(changes in empirical phenomena) , outcome statements (verbal 
expressions) , and outcome measures (observable indicators) . Thl 
methodological framework recognized the need for evaluation 
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standards in order to carry, systematic inquiry- beyond descrip- 
tion and research into the sphere of evaluation proper. Figure 
4 also listed four potential areas in which evaluation find- 
ings might be applied — policy » program design, program manage- 
ment, and accountability — whether for program improvement or 
.other purposes. The importance of formulating outcome state- 
ments explicitly and unambiguously was emphasized both for 
clarity of thinking and communication. An outcome statement or 
question typically identifies an educational treatment, affect- 
ed entity, and specific outcome as well as the agency through 
which the treatment is delivered. An outcome statement may 
also identify a time framework and rationale for the stated 

outcome. ^ " 

o 

Among the key issues identified in the area of outcomes 
evaluation were the definition of vocational edu:ation, the 
specification of vocational programs as educational treat- 
ments (see Figure 5) , sources and uses of evaluative cri- 
teria, experimental controls, and the use of comparison 
groups in evaluating vocational education programs with 
respect to specified outcomes. Table II distinguished fiv^ 
types of data used in outcomes evaluation, classified 
according to the nature of the observation atld methods of 
data collection. A review of outcome studies disclosed 
heavier use of Status (Nonmonetary) and Personal Judgment 
data as opposed to Monetary, Ascriptive, and Performance 
data, with corresponding implications for generalizability , 
credibility, and imp^ict on potential adudiences. 

B. Suggested Agenda for Improving Outcomes Evaluation 

Along with others, we find that empirical research has 
not documented a substantial body of knowledge concerning 

th<*^ outcomes o£ vocational education. Additional evaluation 

studies which repeat shortcomings of design and methodology 
that have been noted will not add significantly to our 
accumulation of knowledge. Some new initiatives are necessary 
to improve the art of outcomes evaluation. 



We therefore recommend the formulation cf a research, 
development, and evaluation (RD&E) agenda, aimed at pro- 
ducing clear, accurate, relevant, credible, and communi- 
cable findings. Salient features of such an agenda would 
be conceptual and empirical answers to such questions as: 



(1) What is vocational education? 

(2) Who are vocational students? 
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(3) What are the outcome goals of vocational 
education? 

(4) What outcomea should be used as criteria to 
evaluate vocational education, and why? 

(5) What data are needed to describe and assess the 
effects of vocational education from the national 

' as well as state and local perspectives, and how 
can such data best be collected and analyzed? 

(6) What standards should be applied to particular 
outcome measures for purposes of evaluation, and 
how should they be determined? 

(7) What specific "input" variables (under the head- 
ings of Student Cnaracteristics, Educational and 
Social Context, Resources, Goals, and Processes) 
in the vocational, education enterprise are signif- 
icantly associated with particular Outcomes, and 
how are they functionally related? 

(8) To what extent do extraneous forces and error 
factors inhibit the production of knowledge 
concerning "the correlates" of particular voca- 
tional education outcomes? 

(9) What evaluation methods and procedures should be 
used to produce accurate and credible general iza- 
tions at the national level regarding the outcomes 
of vocational education? 

(10) How can evaluation findings on vocational educa- 
tion outcomes best be communicated to relevant 
audiences, including policy makers, practitioners, 
and other evaluators to assure that such findings 
will be applied in the interests of improved edu- 
cational ettectiveness and efficiency? 



^^This is the research question currently being ad- 
dressed in the study to which this essay contributes. We 
feel that in its first year, this multi-year project pro- 
gressed about 35%-45% of the way toward developing some 
meaningful and useful answers to the question. During Year 
Two (February 1979 to February 1980) , attention will be 
focused on approximately six to eight specific outcomes- 
designing operational procedures and determining the extent 
to which the selected outcomes and procedures are regarded 
by relevant audiences as appropriate for u&e in evaluating 
vocational programs. 
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The conceptual and empirical answers prescribed for these 
questions must be meaningful and well documented so they 
can serve as a basis for developing authoritative, widely 
shared definitions and standards. Where. value criteria are 
lacking and where empirical evidence is incomplete and 
conceptualizations' imprecise, deficiencies should be openly 
acknowledged and a cpgent rationale provided to justify 
whatever course of action is adopted as an expedient in 
light of the spiecified rational-empirical-normative gaps. 



Action on the recommended RD&E agenda requires a will- 
ingness to acknowledge— with neither defensiveness nor 
resignation— existing shortcomings as well as the strengths 
of vocational education, both with respect to program 
performance and the knowledge we possess about the system's 
components and impact. And the "high tolerance- for ambig- 
uity" that is sometimes counseled for vocational education 
researchers and evaluators needs to be challenged in the 
interests of better evaluation and vocational education's 
commitment to high quality and responsiveness to the needs 
''of individuals and society. 



« 
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